OutOfMemoryError happens when try to decode the wrong ByteArray #2649
Description
Describe the bug
Sometimes our server get wrong ByteArray. We hoped the server throw wrong format error immediately when we try to decode the message. However, protobuf decoder blocks until java heap space out of memory error happens.
From our debugging, message is keep pushBacked by reader.pushBackTag() and index never moves forward from this method.
private fun decodeTaggedListIndex(): Int {
val protoId = if (index == -1) {
// For the very first element tag is already read by the parent
reader.currentId
} else {
reader.readTag()
}
return if (protoId == tagOrSize.protoId) {
++index
} else {
// If we read tag of a different message, push it back to the reader and bail out
reader.pushBackTag()
CompositeDecoder.DECODE_DONE
}
}To Reproduce
// class
@Serializable
@SerialName("FrontRequest")
data class FrontRequest (
val requests: List<String>,
) {
companion object {
@OptIn(ExperimentalSerializationApi::class)
fun fromProtobuf(bytes: ByteArray): FrontRequest
= ProtoBuf.decodeFromByteArray(bytes)
}
}
//test
val payloadBase64 = "+kCbAII+8m6eL8J/nzj6GfwQwzzAa6gzw2uhO8FJ/AGbOv9vr2y/P8l1o1SWPfcylGOnaMFroz7BOak6xGykapQ6qG6Sa/M5lD+jb5RupzvFbvBvxGmnbZVt9WiSaqY6yWL1bpQ/8z3EbvdpwG6pacRj9GrEbfI8xmr1OpM+o2nHPvBqwT7wb8Fup2g="
val payload = Base64.getDecoder().decode(payloadBase64)
//keep working until java heap space error
val frontRequestReceived = FrontRequest.fromProtobuf(payload)
// result
Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3609)
at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.expand(ProtobufTaggedBase.kt:40)
at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.pushTag(ProtobufTaggedBase.kt:34)
at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.access$pushTag(ProtobufTaggedBase.kt:17)
at kotlinx.serialization.protobuf.internal.ProtobufTaggedDecoder.decodeSerializableElement(ProtobufTaggedDecoder.kt:110)
at com.dunamu.quot.realtime.front.request.FrontRequest$$serializer.deserialize(FrontRequest.kt:13)
Expected behavior
throw wrong format error immediately.
Environment
- Kotlin version: 1.8.21
- Library version: 1.5.1
- Kotlin platforms: JVM
- Gradle version: 7.5.1