Strange vertical alignment

If `.xxx-input, .xxx-input-shadow` font-size is changed, the the poistioned :before content moves randomly up/down. Try 19px vs 20px vs 21px font-size

HTML

<div class="input-cover">
  <input class="xxx-input" type="text" inputmode="numeric" value="123">
  <div class="xxx-input-shadow">
    <span class="number">0</span><span class="number">0</span><span class="number">0</span><span class="empty">0</span><span class="empty">0</span>
  </div>
</div>

CSS

body {
    font-family: sans-serif;
}

.input-cover {
  position: relative;
  display: block;
  width: 100%;
  vertical-align: middle;
  border: 1px solid silver;
  padding: 5px 15px;
  box-sizing: border-box;
  background: #fff;
  color: #000;
}

.xxx-input, .xxx-input-shadow {
  margin: 0;
  padding: 0;
  border: 0;
  outline: none;
  font-variant-numeric: tabular-nums;
  font-family: inherit;
  font-size: 64px;
  letter-spacing: .25em;
  line-height: inherit;
  color: inherit;
}
  
.xxx-input {
  position: relative;
  z-index: 1;
  width: 100%;
  background: transparent;
}

.xxx-input-shadow {
  display: block;
  position: absolute;
  z-index: 0;
  left: 15px;
  right: 15px;
  top: 5px;
  bottom: 5px;
  overflow: hidden;
  background: transparent;
  color: #cce;
}

.xxx-input-shadow .empty {
  visibility: hidden;
  position: relative;
}
.xxx-input-shadow .empty::before {
  content: 'E';
  position: absolute;
  left: 0;
  top: 50%;
  transform: translateY(-50%);
  font-size: 0.9em;
  visibility: visible;
  color: orange;
}