Regex is Magic
by Sara Jakša
What the AI was Like at my First Job
The "AI" example
amount:
- Total\s*:\s*\$([\d\.\,]+)
- Total amount\s*:\s*\$([\d\.\,]+)
date:
- (\d+?[st|nd|rd|th]* [A-Z][a-z][a-z], \d\d\d\d)
- (\d\d\/\d\d\/\d\d\d\d)
- ([A-Z][a-z]+?-\d\d-\d\d\d\d)
invoice_number:
- INV(\S+[\d]+)
- Invoice ID:[\s\S]*?(\d+)Regular Expressions
Sequence of characters that specify a search pattern in text.
Example: Bratislava inportance in the last 1000 years
Text from: https://en.wikipedia.org/wiki/Bratislava
\d{4}Bratislava Hist
Bratislava Hist
How regular expessions are marked
The mixture of:
- literal characters -> characters, that represent themsleves only
- meta characters -> representing the groups of characters
Going back to the Bratislava example
\d{4}This regex means to search for 4 characters, that are digits.
Types of special characters for character groups
- \d -> digit
- \s -> space
- \w -> alphanumeric character
- . -> any character
Types of special characters for character groups
- -> 0 or more repetition
- -> 1 or more repetition
- ? -> 0 or 1 repetition or non-greedy variant
- {n} -> number of repetitions
- [] -> any of these characters
- [^] -> none of these characters
- | -> or
Some use cases: Log processing
Fetching data for REGION CUSTOMER (CUSTOMER_ID) - SERVICE failed with status ERROR due to
Fetching data for (\w+) ([\w\s]+?) \((\d+?)\) - ([\w\.]+?) failed with status (.+?) due toRegex Flags
- re.DOTALL -> dot matches also new line
- re.MULTILINE -> ^ and $ match also new lines
- re.ASCII -> match only ASCII
- re.IGNORECASE -> do case-isensitive search
Difference between languages
My name is Sara Jakša
But sometimes this is not a valid name.
Name matching in Python vs JavaScript
Python (valid)
re.findall("[^\w\s]+", "Sara Jakša")JavaScript (not valid)
"Sara Jakša".match(/[^\w\s]+/g)Python ASCII (not valid)
re.findall("[^\w\s]+", "Sara Jakša", flags=re.ASCII)Example: Language learning
48
00:04:03,494 --> 00:04:04,745
（リオン）最高だね
49
00:04:04,828 --> 00:04:05,829
あっ
50
00:04:06,413 --> 00:04:08,916
（リオン）もう 最高の気分だよ！
51
00:04:08,999 --> 00:04:11,251
俺は 確かに傲慢だが—
52
00:04:11,335 --> 00:04:14,588
お前らは そんな俺にも勝てないわけだ
53
00:04:14,672 --> 00:04:19,718
格下に見ていた相手に負ける気分は
どうですか？ 王子様！
54
00:04:19,802 --> 00:04:22,096
き… 貴様！
55
00:04:22,846 --> 00:04:25,849
（リオン）何が
“王族になど生まれたくなかった”だ
56
00:04:26,266 --> 00:04:30,896
お前 変態ババアに売られて
殺されそうになったこと あるのか？
57
00:04:30,980 --> 00:04:31,981
（ユリウス）何!?
58
00:04:32,398 --> 00:04:35,234
女子にペコペコ 頭下げた上に—
59
00:04:35,567 --> 00:04:38,737
お茶会を台なしにされた経験は？
60
00:04:39,113 --> 00:04:43,575
話しかけただけで
突き飛ばされた俺たちの気持ちが—
61
00:04:43,659 --> 00:04:45,744
分かるのかよ！Example: Language learning
data = re.sub(r"^\d+?$", "\n", data, flags=re.M)
data = re.sub(r"\d\d:\d\d:\d\d,\d\d\d --> \d\d:\d\d:\d\d,\d\d\d", "", data)
data = re.sub(r"（[^）]+?）", "", data, flags=re.M)
data = re.sub(r"\\u\d+?", "", data)
data = re.sub(r"\n+", "\n", data)Example: Language learning
最高だね
あっ
もう 最高の気分だよ！
俺は 確かに傲慢だが—
お前らは そんな俺にも勝てないわけだ
格下に見ていた相手に負ける気分は
どうですか？ 王子様！
き… 貴様！
何が
“王族になど生まれたくなかった”だ
お前 変態ババアに売られて
殺されそうになったこと あるのか？
何!?
女子にペコペコ 頭下げた上に—
お茶会を台なしにされた経験は？
話しかけただけで
突き飛ばされた俺たちの気持ちが—
分かるのかよExample: Language learning
import collections
kanji = re.sub(r"[^\u4E00-\u9FFF]", "", data)
len(set(kanji))
# 420
collections.Counter(kanji).most_common(10)
#[('俺', 37), ('殿', 28), ('下', 27), ('気', 24), ('分', 23)]Example: Emoji
import regex
regex.findall("\p{Emoji}", "plenty of 🐟 in the 🌊")
# ['🐟', '🌊']Example: Code rewrite
Example: Code rewrite
csv()
.fromString(csvString)
.on('json', (jsonObject) => {
//something is done
})
.on('done', () => {
//something is done
resolve(result)
});Example: Code rewrite
csv\(\)[\s\n]*?
\.fromString\((.+)\)[\s\n]*?
\.on\(\s*?'json', \(.*?\) => \{([\s\S\n]+?)\}\)[\s\S\n]+
\.on\('done', \(.*?\) => \{([\s\S\n]+?)\}\)Conclusion
Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.
Thank you for listening. Questions?
Contact: sarajaksa@sarajaksa.eu or approach me during this conference
Presentation is available on https://sarajaksa.eu/content/presentations/2022/pycon-slovakia-regex/
Regex is Magic
by Sara Jakša