I have two CSV datasets with the same apparent structure, but one Glue crawler table handles values like 12345,"Smith, John",98765 correctly while the other splits the quoted name at the comma. The broken table uses a basic comma-delimited format and marks columns as unquoted. I have tried recreating the table and adding a CSV classifier. Is there a way to make the crawler use a quote-aware SerDe, or a better way to diagnose its choice?
Why is my Glue crawler splitting quoted CSV fields?
Asked by
7+Quindle
Answers
Check the beginning of the files, especially the header and the first few data rows. Crawlers infer a schema from sampled data, so differences there can affect what they decide. Compare the working and failing files byte for byte around the quoted fields, including whether the quote characters and line endings are consistent.
Reply to this answer
I would not rely on a crawler to preserve a particular SerDe choice. Define the table yourself with the intended schema and OpenCSVSerde, then configure the crawler's schema update behavior so it does not overwrite that definition. If the table is managed as infrastructure, defining it there makes the setting easier to keep consistent.
+3Pipwick:
Recreating the table alone may not help if the crawler keeps inferring the same format. The key is preventing later crawler runs from changing the table definition.
Reply to this answer
Separate the two jobs: use the crawler to discover files or keep track of partitions, but treat the table schema and CSV parsing settings as explicit configuration. Then test a query against a small sample before processing the full dataset. That makes it clearer whether the issue is inference or how the data is being read.
Reply to this answer
Also inspect the raw file rather than a spreadsheet preview. Make sure the embedded comma is actually inside double quotes in the stored file, and check for inconsistent quoting across rows. One malformed or differently formatted sample can make the inferred table a poor fit for the rest of the data.
Reply to this answer
Add your answer
Answers are reviewed before publication.
7+Quindle:
There is a header row. I will compare the first records in both files more closely.