Skip to content

New LSTM error! #56

@liqiangniu

Description

@liqiangniu

When lstm run more than 2000 batches in first iteration, the error (float point exception) occurred! some print info as following:

I0909 17:38:59.281976 23954 TrainerInternal.cpp:162] Batch=2000 samples=128000 AvgCost=0.239344 CurrentCost=0.201116 Eval: classification_error_evaluator=0.0999063 CurrentEval: classification_error_evaluator=0.0798125
I0909 17:40:21.527977 23954 Tester.cpp:111] Test samples=25000 cost=0.184603 Eval: classification_error_evaluator=0.07068
.............................................................................................................................. .........................................................................................................................................................................................................................................................................................................................................................................................................................Current Layer forward/backward stack is
LayerName: lstmemory_0
LayerName: fc_layer_0
LayerName: embedding_0
LayerName: word
*** Aborted at 1473414360 (unix time) try "date -d @1473414360" if you are using GNU date ***
Current Layer forward/backward stack is
PC: @ 0x7f0182f0ab25 __ieee754_exp
Current Layer forward/backward stack is
*** SIGFPE (@0x7f0182f0ab25) received by PID 23954 (TID 0x7f017130c700) from PID 18446744071611394853; stack trace: ***
Current Layer forward/backward stack is
@ 0x7f018414d710 (unknown)
Current Layer forward/backward stack is
@ 0x7f0182f0ab25 __ieee754_exp
Current Layer forward/backward stack is
@ 0x7f0182f20b52 __GI___exp
Current Layer forward/backward stack is
@ 0x800235 hppl::tanh()
Current Layer forward/backward stack is
@ 0x587383 paddle::LstmCompute::forwardOneSequence<>()
Current Layer forward/backward stack is
@ 0x58788a paddle::LstmCompute::forwardBatch<>()
Current Layer forward/backward stack is
@ 0x581bdc paddle::LstmLayer::forwardBatch()
Current Layer forward/backward stack is
@ 0x58521a paddle::LstmLayer::forward()
Current Layer forward/backward stack is
@ 0x614b94 paddle::NeuralNetwork::forward()
Current Layer forward/backward stack is
@ 0x61efe6 paddle::TrainerThread::forward()
Current Layer forward/backward stack is
@ 0x621194 paddle::TrainerThread::computeThread()
Current Layer forward/backward stack is
@ 0x7f01832553d2 execute_native_thread_routine
Current Layer forward/backward stack is
@ 0x7f01841459d1 start_thread
Current Layer forward/backward stack is
@ 0x7f0182a3a8fd clone
/data11/dis_ml/deeplearning/paddle/bin/paddle: line 46: 23954 Floating point exception${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_trainer ${@:2}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions