Skip to content

Csv parser typing #4680

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Nov 3, 2023
Merged

Csv parser typing #4680

merged 6 commits into from
Nov 3, 2023

Conversation

bazsi
Copy link
Collaborator

@bazsi bazsi commented Oct 21, 2023

This PR adds type validation support to csv-parser:

csv-parser(delimiters(',')  columns(int(a), float(b)) dialect(escape-backslash));

Notice the type-hints around the column names. If the type validation fails, the parsing will fail too. What happens is determined by drop-invalid.

This PR depends on #4678

@kira-syslogng
Copy link
Contributor

Build FAILURE

@bazsi bazsi force-pushed the csv-parser-typing branch 2 times, most recently from 66120b6 to 0f67d5d Compare October 22, 2023 13:25
@kira-syslogng
Copy link
Contributor

Build FAILURE

@bazsi
Copy link
Collaborator Author

bazsi commented Oct 22, 2023

@kira-syslogng retest this please;

@kira-syslogng
Copy link
Contributor

Build FAILURE

@bazsi bazsi mentioned this pull request Oct 22, 2023
Merged
@bazsi
Copy link
Collaborator Author

bazsi commented Oct 22, 2023 •
edited by MrAnno
Loading

@kira-syslogng seems to have issues:

17:10:47 GitHub pull request #4680 of commit 0f67d5d6fb6886c1c8132c267b5c8fc57d7839df, no merge conflicts.
17:10:47 Running as SYSTEM
17:10:47 Setting status of 0f67d5d6fb6886c1c8132c267b5c8fc57d7839df to PENDING with url https://ci.syslog-ng.com/job/kira-starter/12636/ and message: 'Build started'
17:10:47 Using context: Kira-starter
17:10:48 [EnvInject] - Loading node environment variables.
17:10:48 Building remotely on [172.31.254.20](https://ci.syslog-ng.com/computer/172.31.254.20) (slave) in workspace /mnt/jenkins/workspace/kira-starter
17:10:48 The recommended git tool is: NONE
17:10:48 No credentials specified
17:10:48  > git rev-parse --resolve-git-dir /mnt/jenkins/workspace/kira-starter/syslog-ng-ose/.git # timeout=10
17:10:48 Fetching changes from the remote Git repository
17:10:48  > git config remote.origin.url https://github.com/syslog-ng/syslog-ng.git # timeout=10
17:10:48 Fetching upstream changes from https://github.com/syslog-ng/syslog-ng.git
17:10:48  > git --version # timeout=10
17:10:48  > git --version # 'git version 2.17.1'
17:10:48  > git fetch --tags --progress -- https://github.com/syslog-ng/syslog-ng.git +refs/pull/*:refs/remotes/origin/pr/* # timeout=10
17:10:48 ERROR: Error fetching remote repo 'origin'
17:10:48 hudson.plugins.git.GitException: Failed to fetch from https://github.com/syslog-ng/syslog-ng.git
17:10:48 	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:999)
17:10:48 	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1241)
17:10:48 	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1305)
17:10:48 	at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
17:10:48 	at hudson.scm.SCM.checkout(SCM.java:540)
17:10:48 	at hudson.model.AbstractProject.checkout(AbstractProject.java:1245)
17:10:48 	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
17:10:48 	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
17:10:48 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
17:10:48 	at com.tikal.jenkins.plugins.multijob.MultiJobBuild$MultiJobRunnerImpl.run(MultiJobBuild.java:148)
17:10:48 	at hudson.model.Run.execute(Run.java:1900)
17:10:48 	at com.tikal.jenkins.plugins.multijob.MultiJobBuild.run(MultiJobBuild.java:76)
17:10:48 	at hudson.model.ResourceController.execute(ResourceController.java:101)
17:10:48 	at hudson.model.Executor.run(Executor.java:442)
17:10:48 Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/syslog-ng/syslog-ng.git +refs/pull/*:refs/remotes/origin/pr/*" returned status code 128:
17:10:48 stdout: 
17:10:48 stderr: fatal: unable to access 'https://github.com/syslog-ng/syslog-ng.git/': gnutls_handshake() failed: Error in the push function.

@MrAnno
Copy link
Collaborator

MrAnno commented Oct 23, 2023

@kira-syslogng retest this please

@kira-syslogng
Copy link
Contributor

Build FAILURE

@bazsi
Copy link
Collaborator Author

bazsi commented Oct 24, 2023

@kira-syslogng retest this please;

bazsi added 6 commits October 24, 2023 11:54
@bazsi
msg-generator: add support for typed values
c79cea0 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
cfg-grammar: extract type_hint parsing from template_content_inner
47274ce 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
csv-parser: extract _iterate_columns() function
6d48e42 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
add-contextual-data: delay the freeing of the literal template string
437a6fa 
We are freeing the original template string that we've read from the
database, as it can consume a lot of memory with large databases.

Originally, we did this right after instantiating the template.
This is earlier than the time we report that the database record
was added.

By delaying the call to log_template_forget_template_string() we get the
proper value of the template when we report it in our log message.

Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
csv-parser: move the list of columns from CSVScanner to CSVParser
e3a3523 
This patch moves the notion of "column names" from CSVScanner to CSVParser,
as a preparatory step of adding type validation to CSVParser.

Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
csv-parser: add support for type validation of columns
b37209d 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi bazsi force-pushed the csv-parser-typing branch from 0f67d5d to b37209d Compare October 24, 2023 09:54
@MrAnno MrAnno self-requested a review October 25, 2023 21:50
@MrAnno MrAnno merged commit debe6b6 into syslog-ng:master Nov 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
1 more reviewer

@MrAnno MrAnno MrAnno approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@bazsi @kira-syslogng @MrAnno