runtime.boundsError when trying to access index on memvaruint

[pier-oliviert]

Here's the stack trace of the most recent event:

runtime.boundsError runtime error: index out of range [5250] with length 5250 
    runtime/panic.go:88 goPanicIndex
    github.com/blugelabs/ice@v0.1.1/memuvarint.go:64 (*memUvarintReader).ReadUvarint
    github.com/blugelabs/ice@v0.1.1/intdecoder.go:118 (*chunkedIntDecoder).readUvarint
    github.com/blugelabs/ice@v0.1.1/posting.go:387 (*PostingsIterator).skipFreqNormReadHasLocs
    github.com/blugelabs/ice@v0.1.1/posting.go:644 (*PostingsIterator).currChunkNext
    github.com/blugelabs/ice@v0.1.1/posting.go:562 (*PostingsIterator).nextDocNumAtOrAfter
    github.com/blugelabs/ice@v0.1.1/posting.go:458 (*PostingsIterator).nextAtOrAfter
    github.com/blugelabs/ice@v0.1.1/posting.go:451 (*PostingsIterator).Advance
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/index/postings.go:100 (*postingsIterator).Advance
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/search/searcher/search_term.go:97 (*TermSearcher).Advance
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/search/searcher/search_disjunction_slice.go:199 (*DisjunctionSliceSearcher).Advance
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/search/searcher/search_disjunction_heap.go:197 (*DisjunctionHeapSearcher).Advance
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/search/searcher/search_boolean.go:340 (*BooleanSearcher).advanceIfTrailing
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/search/searcher/search_boolean.go:301 (*BooleanSearcher).Advance
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/search/searcher/search_conjunction.go:207 (*ConjunctionSearcher).advanceChild
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/search/searcher/search_conjunction.go:153 (*ConjunctionSearcher).Next
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/search/searcher/search_boolean.go:81 (*BooleanSearcher).initSearchers
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/search/searcher/search_boolean.go:156 (*BooleanSearcher).Next
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/search/searcher/search_conjunction.go:96 (*ConjunctionSearcher).initSearchers
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/search/searcher/search_conjunction.go:107 (*ConjunctionSearcher).Next
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/search/searcher/search_boolean.go:81 (*BooleanSearcher).initSearchers
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/search/searcher/search_boolean.go:156 (*BooleanSearcher).Next
    /go/pkg/mod/github.com/blugelabs/bluge@v0.1.5/search/collector/all.go:77 (*AllIterator).Next
    /app/nodes/filters/filter.go:85 (*Index).SubscriberIds
    /app/nodes/controllers/subscriber_controller.go:47 (*SubscriberController).Filter
    /app/protos/services/services_grpc.pb.go:116 _Subscribers_Filter_Handler.func1
    /app/interceptors/requestid.go:23 CreateRequestIDInterceptor.func1
    github.com/grpc-ecosystem/go-grpc-middleware@v1.2.0/chain.go:25 ChainUnaryServer.func1.1.1
    /app/interceptors/stg.go:38 CreateSTGInterceptor.func1
    github.com/grpc-ecosystem/go-grpc-middleware@v1.2.0/chain.go:25 ChainUnaryServer.func1.1.1
    github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107 (*ServerMetrics).UnaryServerInterceptor.func1
    github.com/grpc-ecosystem/go-grpc-middleware@v1.2.0/chain.go:25 ChainUnaryServer.func1.1.1
    /app/interceptors/bugsnag.go:38 bugsnagConfiguredInterceptor
    github.com/grpc-ecosystem/go-grpc-middleware@v1.2.0/chain.go:25 ChainUnaryServer.func1.1.1
    go.elastic.co/apm/module/apmgrpc@v1.8.0/server.go:91 NewUnaryServerInterceptor.func1
    github.com/grpc-ecosystem/go-grpc-middleware@v1.2.0/chain.go:25 ChainUnaryServer.func1.1.1
    github.com/grpc-ecosystem/go-grpc-middleware@v1.2.0/chain.go:34 ChainUnaryServer.func1
    /app/protos/services/services_grpc.pb.go:118 _Subscribers_Filter_Handler
    google.golang.org/grpc@v1.34.0/server.go:1210 (*Server).processUnaryRPC
    google.golang.org/grpc@v1.34.0/server.go:1533 (*Server).handleStream
    google.golang.org/grpc@v1.34.0/server.go:871 (*Server).serveStreams.func1.2
    runtime/asm_amd64.s:1373 goexit

I don't have more much to add to this right now, I'll add more content to this ticket as I investigate and gather more information.

[pier-oliviert]

Most likely an OOM crash that corrupted the file. Closing for now.

[mschoch]

Interesting, if OOM killed us while in the middle of writing a segment, we should not have any valid snapshots pointing to it yet. I'm going to leave this open for now...