Structuring Machine Learning Project

week1

**WARP UP

1. Avoidable error

high bias problem

- bigger model, better optimization, architecture, hyperparameter

high variance   problem

- more training, dev data set, regularization, architecture, hyperparameter

 

* 개발 step별 개선 방법

training set -> dev set -> test set -> real world

a. training set : high bias problem /// human-level과 비교시 error 차이가 큰경우

 - bigger model, another optimization // 모델의 복잡도 개선, cost를 줄이는 방법 변경(Adam optimization)

b. dev set : high variance problem /// train 과 dev의 error차이가 큰경우

 - regularization, bigger training set  //  overfitting 개선 - dropout, early stopping, / agumentation 

c. test set : difference of distribution between dev and test set /// data의 분포가 다름

 - bigger dev set

d. real world : similar with above problem(difference of distribution) /// bigger dev set

 

* evaluation metric

must use a single number evaluation metric // precision, accuracy, f1 score등 한가지 metric을 사용할것

type of metric : optimizing, satisficing  // 최적화해야할 요소(정확도 등)와 만족만 하면 되는 요소(메모리 사용량 등)

 

* orthogonality 

worry seperately between 1. selecting right evaluation metric and 2. way of optimizing cost function 

metric 선정과 optimizing을 개별로 선정하고 개선한다

 

* problem in applying of machine to real world

difference between dev set and real world //  개발 초기에 적절한 objec(cost)와 data 분포를 고려한다 

dev set : high quality, resolution image

real world : blurred, distorted, tilted image

 

* avoidable error // 목표 성능을 결정하는 지표(사람 수준)를 활용해서 bias, variance를 구분짓고 학습 방향을 결정한다

set human-level performance(HLP) as prox for baye optimal error

gap of HLP with dev error, test error is to be reduced during learning

distingush with high bias, variance using avoidable error 

 

* surpass human-level performance

 

week2

**WARP UP

1. Error Analysis (avoidable error)

Avoidable error, ceiling analysis, mismatched data

- avoidable error를 찾고, ceiling analysis를 통해서 개선할 단계들의 순위를 정하여 실행 또는 무시할지 결정한다

- train-dev-test set에서의 성능/error rate를 비교하면서 error 의 종류를 확인하고 그에 맞는 전략으로 대응한다

(avoidable error, variance, data mismatched 구별)

2. Transfer learning(pre-trained model)

General feature를 base로 Special feture로 확장시킨다

(low level -> high level // object recognition -> cancer recognition)

 

* ceiling analysis

 find factor maximizing accuracy in each step

 make sure it's worth fixing incorrectly labeled data in train set, test set 

 error의 원인중 가장 비중이 높은 원인부터 해결, 개선의 효과가 적은것은 순위에 밀림, incorrected label, ohters...

 

*mismatched data

 when correcting, work to have same distribution on train, dev, test set  (they are equally checked)

오라벨을 수정할때 test set과 train set, dev set 모두 확인해야한다 

train set은 웹 사진, test set은 모바일 사진으로 사용하면 두 set의 distribution은 다른것이다 

differece distribution of data set(train, dev, test) makes higher error in test set 

	General speech recogniton	Special speech recognition
	Human-level 4%	6%
Avoidable Error	raining error 7%	6%
Variance	training-dev error 10%	dev-test error 6%
		Data mismatched

Avoidable error// variance// data mismatched (small data ->> using artificial synthesis reflecting whole data set)

small data는 agumentation으로 일부 해결 가능

 

* first build and iterately modify

1. set up data set   2. set metric  3. build intial system quickly  4. analysis bias/variance/error 5. modify system

 

* transfer learning(pre-trained model)

use general model first, and chanege classification layer to new layer for learning specific feature.

then layer of the first model be frozened

depending on the case of small or big data set, choose size of layer to be frozen

 - VGG, Inception, RecNet5 ...