Adds Incapsula log format #184
Conversation
|
Sorry to get back to you about this again so soon.
|
|
Ok. Having them blank won't with the regex. But that should be easy to fix. I've pushed an update. Would you mind to test it again? |
|
Just tested it with the log above and unfortunately it doesen't work. Unless each log entry is populated with data in the status and length fields the script will hang forever. Oncce populated the import goes through without any problems.
|
|
@dinasty02091994 please try again, hope it is fixed now. |
|
@sgiehl I just tried with the updates and it was still hanging the same way as before. I made the following change to the code to make it work: Before the change Line 362: after the change line 362: I think this has to do with some requests being made directly to the topsite without any slashes which made the script hang. This seemed to be due to the regex looking for atleast one slash in the path field. Not sure if this was the correct way to solve it but i thought you should know about it anyways. Now all my logs imports the way they should without any problems :) |
|
Good catch. Will change that. |
|
I also made the following change. I noticied that the script continued hanging on some logs while others worked fine. I analyzed what was wrong and got to the following conclusion: Since the path field sometimes contain non standard characters ( swedish website ) i had to change the following: Line 362 before change:
Line 362 after change:
After this change i've imported well over 1000 logfiles without any problem at all. |
|
@dinasty02091994 we are using ´\S´ almost everywhere to match the strings. Reason why it doesn't match in your case might be the locale used on your system. |
|
Hello Just one mention regarding this part is what incapsula support different log formats. None worked before. The format this is more specific for is Incapsula method of W3C. They support CEF and LEEF to but the fixes here is done for Incapsula W3C. So maybe the name of the log-format-name should be incapsula_w3c ? Regards |
|
@magnus-84 thx for your note. I'll change the name. Do you have log examples for the other formats? Can the other format types also be used to be imported to Piwik? If so, do you maybe have example logs? Maybe we could add them later as well. |
|
@dinasty02091994 do you have an example log line that wasn't imported cause of unmatched characters? |
|
@sgiehl Here is some example of CEF and LEEF logs. Let me know if you need more. I can´t test it since we are not currently using it. It´s parts of logs from when we tested going in to production. Incapsula CEF format example log CEF:0|Incapsula|SIEMintegration|1|1|Normal|0| fileId=262000500012870279 sourceServiceName=www.example.se siteid=44850949 suid=774502 requestClientApplication=Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 6.0) cs2=false cs2Label=Javascript Support cs3=false cs3Label=CO Support src=123.123.123.123 cs1=NA cs1Label=Cap Support cs4=fe747c2f-3f46-45ba-97e1-efd3f8cc8646 cs4Label=VID cs5=22c153a569931a8adb0730d147007186a16be1aa51cc9a296406f116cdf401ba466691f0bb5c14c82387f7faed8e892dfc712bcbb2559ef868a34e36f2904c61 cs5Label=clappsig dproc=Unclassified cs6=Bot cs6Label=clapp ccode=[US] tag=LK cicode=Mountain View cs7=37.4192 cs7Label=latitude cs8=-122.0574 cs8Label=longitude Customer=Examplecust start=1498509920937 request=www.example.se/rss/feed/79539 requestMethod=GET cn1=301 app=HTTP act=REQ_PASSED deviceExternalId=20679399964017444 sip=234.234.234.234 spt=80 in=519 xff=123.123.123.123 Incapsula LEEF format example log Link to beter formated logs. |
Incapsula logs won't be auto detected as they break with the other detections
Based on #180
Fixes #179