Using Custom Audience usersreplace with multiple schema, but not multi-key

I have a Custom Audience that's being generated from a customer list. That customer list is composed of both customer's phone numbers and emails, but not necessarily both a phone number and an email linked to a single customer, so I don't want to use multi-key matching. I want to use the usersreplace endpoint to update this list on a regular basis. I want to go through all the phones and add them in a single session and then go through all the emails and add them in that same session.
I am able to successfully add all of the first schema type. But when I get to the second schema type, I get a (#2650) service error. I believe this is due to using a second, different schema type, because when I switch to use the email first, I get the same error when I start trying to add the phones.