1. Logistic Regression
1.1. Theory
One sample: Loss function:
\[L(\hat{y},y)=-(ylog(\hat{y}+(1-y)log(1-\hat{y})))
\]
All training samples: Cost function
\[J(w,b)= -\frac 1m \sum_{i=1}^{m}[y^{(i)}log(\hat{y}^{(i)}) + (1-y^{i})log(1-\hat{y}^{(i)})]
\]
1.2. Gradient Descent
One sample: Derivative of the LR:
\[dz^{(i)} = a^{(i)} - y^{(i)} \\
dw_{j}^{(i)} = \frac{\delta L}{\delta w_{j}^{(i)}} = x_{j}^{(i)} \cdot dz^{(i)}(j=1,2,...,n_{x}) \db^{(i)} = dz^{(i)}
\]
All training samples: Derivative of the Cost function:
\[\delta w_{j}J(w,b) = \frac 1m \sum_{i=1}^{m} \frac{\delta}{\delta w_{j}}L(a^{(i)},y^{(i)})
\]
1.3. Implementation
搭建LR的分类器
import numpy as np
class LogisticRegression:
def __init__(self, n_iters = 50, learning_rate = 0.001):
self.epoch = n_iters
self.lr = learning_rate
self.weights = None
self.bias = None
def fit(self,X,y):
# X: input data (m,n)
# y: output label (m,1)
# weights: weight (n,1)
# bias: na
# init parameter
n_samples, n_features = X.shape
self.weights = np.random.rand(n_features)
self.bias = 0
# Gradient descent
for _ in range(self.epoch):
linear_value = np.dot(X, self.weights) + self.bias
y_predicted = self._sigmoid(linear_value)
# compute the derivative
dw,db = self._derivative(X,y,y_predicted,n_samples)
# update the value
self.weights -= self.lr * dw
self.bias -= self.lr * db
def predict(self, X):
# the weight and bia has already been updated
linear_value = np.dot(X, self.weights) + self.bias
y_predicted = self._sigmoid(linear_value)
# compute the predicted class
y_predicted_class = [1 if i>0.5 else 0 for i in y_predicted]
return y_predicted_class
def _sigmoid(self,x):
return 1/(1+np.exp(-x))
def _derivative(self, X, y, y_predicted, n_samples):
dz = y_predicted - y
dw = (1/n_samples)*np.dot(X.T, dz) # (n,m) * (m,1) = (n,1)
db = (1/n_samples)*np.sum(dz)
return dw, db
Test
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn import datasets
import matplotlib.pyplot as plt
from Library import LogisticRegression
data = datasets.load_breast_cancer()
X,y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2, random_state=1234)
def accuracy(y_true,y_pred):
return np.sum(y_true == y_pred) / len(y_true)
regressor = LogisticRegression(learning_rate=0.0001, n_iters=1000)
regressor.fit(X_train, y_train)
predictions = regressor.predict(X_test)
print("LR Classification accuract:", accuracy(y_test,predictions))
最终的结果达到了90%以上的准确度。