Solid-state polymer electrolytes (SPEs), such as poly(ethylene oxide) (PEO), have good flexibility when compared to ceramic-type solid electrolytes. Therefore, it could be an ideal solid electrolyte for zero-excess all-solid-state Li metal battery (ZESSLB), also known as anode-free all-solid-state Li battery, development by offering better contact to the Cu current collector. However, the low Coulombic efficiencies observed from polymer type solid-state Li batteries (SSLBs) raise the concern that PEO may consume the limited amount of Li in ZESSLB to fail the system. Here, we designed ZESSLBs by using all-ceramic half-cells and an extra PEO electrolyte interlayer to study the reactivity between PEO and freshly deposited Li under a real battery operating conduction. By shuttling active Li back from the anode to the cathode, the PEO SPEs can be separated from the ZESSLBs for experimental studies without the influence from cathode materials or possible contamination from the usage of Li foil as the anode. Electrochemical cycling of ZESSLBs shows that the capacities of ZESSLBs with solvent-free and solvent-casted PEO SPEs significantly degraded compared to the ones with Li metal as the anode for the all-solid-state Li batteries. The fast capacity degradation of ZESSLBs using different types of PEO SPEs is evidenced to be associated with Li reacting with PEO, residual solvent, and water in PEO and dead Li formation upon the presence or absence of residual solvent. The results suggest that avoiding direct contact between the PEO electrolyte and deposited lithium is necessary when there is only a limited amount of Li available in ZESSLBs.