perf: Fix issues with Chart|LayerChart.encode, 1.32x speedup to infer_encoding_types
#2700
| Job | Run time |
|---|---|
| 2m 15s | |
| 2m 30s | |
| 3m 3s | |
| 3m 11s | |
| 2m 49s | |
| 3m 2s | |
| 3m 18s | |
| 3m 7s | |
| 23m 15s |