Iso 8601 on vxlabs

PySpark timezone offset from ISO 8601 without UDF

Tue, 15 Oct 2024 19:39:40 +0200

Background

During my work at Stone Three upgrading some of our data pipelines with Azure DataBricks, I was quite disappointed to learn that Spark 3.5.3’s only timezone-capable timestamp type always stores timestamps converted to UTC, and always displays these timestamps converted to the session-global timezone.

Admittedly, the situation is equally bad with PostgreSQL’s TIMESTAMPTZ type, leading to designs where the actual timezone offset must be stored as a separate timezone offset column.

Having a timezone-aware timestamp type that could store the input timezone natively, and which would always display with its full timezone information, would have been much more useful. In our case, we not only need to know the exact timepoint that one of our vision sensors generated a measurement, but we also need to know what the exact local time was for that specific measurement.

Conservative rendering and liberal parsing of ISO 8601 timestamps in Python

Fri, 05 Aug 2022 15:28:00 +0200

You’ll see in Python bug 37962 that datetime.isoformat() and datetime.fromisoformat() use a restricted subset of the ISO 8601 timestamp formatting standards.

This is similar to the subset followed by W3C.

Recently, we ran into the specific issue that fromisoformat() rejects inputs where the timezone is specified as +hhmm instead of +hh:mm.

Both are allowed by ISO 8601, but only the latter is part of the restricted subset.

Following the Robustness principle (also known as Postel’s law) in the design of our APIs, we would prefer to

Convert dates to different formats in LibreOffice Calc

Mon, 06 Oct 2014 12:51:00 +0000

I’m helping someone process a collection of research data that has been entered by a third party using Excel. We’re using LibreOffice Calc, because research should be reproducible by anyone, not just those in possession of prioprietary software licenses (this also means that we use R, JGR and DeduceR instead of SPSS and Statistica; perhaps more on that later).

After having to fix hundreds of badly entered dates with basic functions (we highly recommend that ISO 8601 format dates, i.e. YYYY-MM-DD, are used from the very start, instead of ambiguous local formats), we ended up with a stubborn subset of DD-MM-YYYY formatted dates in cells that explicitly had ISO 8601 format configured.